Skip to content

docs: surface real-provider E2E testing - #123

Draft
ClaudiaFang wants to merge 14 commits into
test/real-provider-e2efrom
docs/e2e-testing-docs
Draft

docs: surface real-provider E2E testing#123
ClaudiaFang wants to merge 14 commits into
test/real-provider-e2efrom
docs/e2e-testing-docs

Conversation

@ClaudiaFang

@ClaudiaFang ClaudiaFang commented Aug 13, 2026

Copy link
Copy Markdown
Member

Summary

  • surface the real-provider E2E strategy in the README
  • add CONTRIBUTING.md with project testing expectations for contributors
  • replace the obsolete temporary E2E note in docs/test-coverage.md with the current layered test model
  • add docs/testing/test-scenarios.md as a behavior- and risk-oriented overview
  • add docs/testing/test-matrix.md with P0/P1/P2 classification and current coverage by test layer
  • document the planned GitHub packaged-plugin black-box P0 suite as a coverage gap and future refactoring guardrail

Why

The repository already has meaningful unit/integration and real-provider E2E coverage, but that quality was difficult to understand from the main documentation. These docs separate four different questions:

  • what behaviors and failure modes the project protects
  • which scenarios are release-critical (P0) vs important/supporting (P1/P2)
  • where the implementation-level tests live
  • how the real-provider E2E harness and CI operate

The current real-provider SyncManager suite is documented accurately as gray-box system/E2E coverage because it imports production internals and replaces the Obsidian boundary. A future packaged-plugin black-box suite is explicitly marked Planned, with GitHub as the canonical P0 user journey target. Nothing in this PR claims that black-box coverage already exists.

The wording also intentionally says the real-provider harness supports GitHub, GitLab, and Gitea rather than claiming all three provider legs are currently stable in CI. Current runner-specific limitations remain documented in docs/testing/real-provider-e2e.md.

Validation

  • documentation-only change
  • reviewed against the current test/real-provider-e2e branch
  • test matrix cross-checked against current SyncManager, GitHub, GitLab, and Gitea E2E suites
  • planned black-box and compatibility coverage clearly separated from implemented coverage
  • final diff reviewed for accidental unrelated changes

ClaudiaFang and others added 14 commits August 13, 2026 03:15
Real-provider E2E returns after the scanner-driven removal in main
(002000e), rebuilt so no committed .ts uses the flagged APIs
(fetch/globalThis/node:crypto/node:child_process/node:util/bare timers),
regardless of directory:

- scripts/e2e-harness.sh (provision/seed/verify/cleanup/sweep): Shell + Git
  CLI owns branch/container lifecycle. GitHub/GitLab isolation via
  `git push <sha>:refs/heads/<branch>`, no REST branch-creation calls.
  Gitea's disposable container+repo via plain docker/curl, never
  node:child_process. GIT_ASKPASS generated per-run under
  $RUNNER_TEMP/$E2E_WORKDIR, never persisted (no token in remote URLs,
  .git/config, credential.helper, args, or logs).
- Node-only glue the suites still need at runtime (requestUrl shim, window
  timer alias, a git-CLI-backed verifier) is generated by `provision` into
  $E2E_RUNTIME_DIR, never committed -- suites import only a type-only
  contract (e2e/verifier-runtime-types.ts) statically and load the concrete
  implementation via a runtime-computed dynamic import(), so npm run
  build's typecheck never needs the harness to have run first.
- Ported all four suites (github/gitlab/gitea/sync-manager) to the unified
  SyncManager.pushFiles API from claude/unify-push-pull-pipeline.
- scripts/run-e2e.sh: local orchestration wrapper (provision -> seed ->
  vitest -> cleanup). CI drives the same steps directly per job step.
- Removed e2e/provision, e2e/verifier/{github,gitlab,gitea}-verifier.ts,
  e2e/providers, e2e/shim/{obsidian-request-url,window-timers}.ts,
  e2e/namespace.ts, e2e/redact.ts, scripts/run-e2e*.mjs,
  scripts/e2e-sweep-branches.mjs -- superseded by the above.
- e2e/**/*.ts back in tsconfig.json's include and eslint's scope.

Verified with a real end-to-end run against a live local Gitea sandbox
(npm run test:e2e -- --provider gitea): 14/14 E2E tests passed, including
a real Docker provision/seed/cleanup cycle. GitHub/GitLab legs are written
and typecheck/lint clean but unverified live (no sandbox credentials in
this environment) -- see docs/testing/real-provider-e2e.md.

npx eslint . -- 0 errors
npm run build -- clean (incl. Obsidian 1.11.0 compat typecheck)
npx vitest run -- 527 passed

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The pushed Phase 1 harness failed its first real CI run against
firstsun-dev/git-files-sync (run 31665711682). Root causes, all found by
reading the actual job logs:

- github/gitlab legs: the vitest step's generated GitVerifier shells out to
  git, but GIT_ASKPASS/GIT_TERMINAL_PROMPT only ever existed inside the
  provision/seed/cleanup steps' own processes -- the vitest step is a
  separate process that only sources e2e.env, which never carried them.
  `git fetch` prompted for a username and failed. Now persisted (as a path,
  not a secret -- the token itself stays only in the mode-700 askpass file
  on disk) in e2e.env's write_env_file/load_env_file.

- gitea leg: provisioning timed out waiting on `127.0.0.1:<host-port>` --
  this runner fleet is itself a sibling container of the Docker daemon, so a
  published host port is only reachable from the Docker host's own network
  namespace, not from a sibling container's. Switched to the gitea
  container's own bridge IP (reachable from any container on the same
  default Docker network, including a sibling runner), dropping the -p
  mapping entirely.

- gitea leg's cleanup step then also failed: cmd_cleanup called
  setup_askpass unconditionally before branching on provider, but gitea's
  cleanup is pure `docker rm` and needs no git credentials -- and since
  provision had already failed before provisioning a token, there was
  nothing for setup_askpass to require. Gitea's branch now runs first and
  skips setup_askpass entirely.

Verified with another real end-to-end run against a live local Gitea
sandbox (npm run test:e2e -- --provider gitea): 14/14 passed, using the
container's bridge IP this time. Full gate still green: eslint 0 errors,
build clean, vitest 527/527.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The github/gitlab-fix push (1393956) triggered a second real CI run: both
github and gitlab legs passed this time, confirming the GIT_ASKPASS/
GIT_TERMINAL_PROMPT propagation fix. The gitea leg hung for 10+ minutes on
"Provision isolated branch/container" -- well past the 60s ready_ms budget
-- and had to be cancelled manually.

Root cause: none of the curl calls in provision_gitea_container had a
--max-time. A curl against an unreachable/blackholed address (e.g. an empty
container_ip if `docker inspect` raced the container's network attachment)
can hang far longer than the health-check loop's own timeout budget,
instead of failing fast into the next retry -- the loop's `waited -ge
ready_ms` check never gets a chance to fire if a single curl call itself
never returns.

Fixes: retry docker inspect up to 10x/1s if container_ip comes back empty
before ever starting the health loop (fail fast with a clear error if it
never does); --max-time on every curl call in this function (5s for the
per-poll healthz check, 15s for the one-shot repo/token/user setup calls
and the gitlab project-lookup call in normalize_env).

Verified locally again (timed): full provision -> seed -> vitest -> cleanup
in 13.6s, no hangs. Full gate still green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The gitea leg still needs more investigation against this specific
self-hosted runner fleet's Docker topology (bridge-IP reachability, health-
check timing already needed two rounds of fixes) -- not something safe to
keep iterating on inside the shared provider-e2e matrix while github/gitlab
are otherwise green. Gate it off via the existing per-provider "Determine
whether this leg should run" step rather than removing it from the matrix,
so job structure/naming stays stable for whoever re-enables it.

Suite and harness code (e2e/suites/gitea.e2e.test.ts, scripts/e2e-
harness.sh's gitea path) is untouched -- verified locally again just now
(`npm run test:e2e -- --provider gitea`, 14/14 passed, 13.6s) -- only CI
execution is paused pending runner-environment follow-up. Re-enable by
deleting the added `if` block once confirmed.

Note: gitea is normally what covers fork PRs without needing real
credentials -- while disabled, fork PRs get zero E2E coverage. Acceptable
short-term given this branch has no open fork PRs yet.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@ClaudiaFang
ClaudiaFang force-pushed the test/real-provider-e2e branch from ec665dc to c42fa35 Compare August 13, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant